Legislative Audit Division 




State of Montana 



Report to the Legislature 



June 2006 



Performance Audit 



The Collection and Use of 
Graduate and Dropout Data 

Office of Public Instruction 

This performance audit provides the Legislature information about the 
collection and use of graduate and dropout data. To ensure data is 
consistent among schools for reporting and decision-making, this report 
makes recommendations for implementing controls as well as a means to 
utilize the data. A written response from the Office of Public Instruction is 
included at the end of the report. 



06P-11 



Direct comments/inquiries to: 
Legislative Audit Division 
Room 160, State Capitol 
PO Box 201705 
Helena MT 59620-1705 



Help eliminate fraud, waste, and abuse in state government. Call the Fraud Hotline at 
1-800-222-4446 statewide or 444-4446 in Helena. 



PERFORMANCE AUDITS 



Performance audits conducted by the Legislative Audit Division are designed to assess state 
government operations. From the audit work, a determination is made as to whether agencies and 
programs are accomplishing their purposes, and whether they can do so with greater efficiency 
and economy. The audit work is conducted in accordance with audit standards set forth by the 
United States Government Accountability Office. 

Members of the performance audit staff hold degrees in disciplines appropriate to the audit 
process. Areas of expertise include business and public administration, mathematics, statistics, 
economics, political science, criminal justice, computer science, education, and biology. 

Performance audits are performed at the request of the Legislative Audit Committee which is a 
bicameral and bipartisan standing committee of the Montana Legislature. The committee consists 
of six members of the Senate and six members of the House of Representatives. 



MEMBERS OF THE LEGISLATIVE AUDIT COMMITTEE 


Senator Joe Balyeat, Vice Chair 


Representative Dee Brown 


Senator John Brueggeman 


Representative Hal Jacobson 


Senator Jim Elhott 


Representative Christine Kaufmann 


Senator Dan Harrington 


Representative Scott Mendenhall 


Senator Lynda Moss 


Representative John Musgrove, Chair 


Senator Corey Stapleton 


Representative Janna Taylor 



LEGISLATIVE AUDIT DIVISION 



Scott A. Seacat, Legislative Auditor 

Tori Hunthausen, 

Chief Deputy Legislative Auditor 




Deputy Legislative Auditors: 

James Gillett 

Jim Pellegrini 



June 2006 



The Legislative Audit Committee 
of the Montana State Legislature: 

This performance audit provides the Legislature information about the collection and use of graduation and 
dropout data. To ensure data is consistent among schools for reporting and decision-making, this report 
makes recommendations for implementing controls as well as a means to utilize the data. A written response 
from the Office of Public Instruction is included at the end of the report. 

We wish to express our appreciation to the Office of Public Instruction staff and local schools for 
their cooperation and assistance during the audit. 

Respectfully submitted, 

/s/ Scott A. Seacat 

Scott A. Seacat 
Legislative Auditor 



Room 160, State Capitol Building, PO Box 201705, Helena, MT 59620-1705 
Phone (406) 444-3122 FAX (406) 444-9784 E-MailladCs'mt.gov 



Legislative Audit Division 



Performance Audit 



The Collection and Use of 
Graduate and Dropout Data 



Office of Public Instruction 



Members of the audit staff involved in this audit were Bridget Butler and 
Angle Grove. 



Table of Contents 



List of Figures and Tables ii 

Elected, Appointed and Administrative Officials iii 

Report Summary S-1 

Chapter I - Introduction and Background 1 

Introduction 1 

Audit Scope and Objectives 1 

Background 2 

State Requirements 2 

Federal Requirements 2 

Calculations of Graduate Data 3 

Calculations of Dropout Data 5 

National Attention 7 

Report Organization 8 

Chapter II - OPI GuideHnes 9 

OPI Guidelines for Reporting 9 

School Visits 10 

Data Verification 11 

Examples of Inconsistency 11 

Guidelines Not Fully Utilized 12 

A Quality Assurance System Needed 12 

Conclusion 13 

Documentation Needed for District Approved GEDs and Home 

School Students 13 

Home School Students 13 

District Approved GED Recipients 14 

Chapter III - OPI's Plan For Improving Data Collection and Accuracy 15 

Initial Planning 15 

Resources Appropriated 15 

Current Stage of Implementation 15 

The New System 16 

Majority of States Have a Statewide Information Management 

System 17 

Impacts of the New System 18 

Human Element Still Exists 18 

Chapter IV - Use of Data 19 

Data is Integral to Decision-Making 19 

National Use of Data for Economic Analysis 19 

Stakeholders' Use of Data 20 

Accessibility and Use of Data 21 

Conclusion 22 

Appendix- A A-1 

Agency Response B-1 



Pagei 



List of Figures and Tables 



Figures 

Figure 1 Economic Benefits of Education in Montana 20 

Tables 

Table 1 National Methods for Calculating Graduation Rates 4 

Table 2 Methods for Calculating Dropout Rates 6 

Table 3 Student Status Definitions 10 



Page ii 



Elected, Appointed and Administrative Officials 



Office of Public Instruction Linda McCulloch, Superintendent 

Dennis "Bud" Williams, Deputy Superintendent 
Nancy Coopersmith, Assistant Superintendent 
Madalyn Quinlan, Chief of Staff 



Donna O'Neill, Division Administrator, 
Measurement and Accountability Division 



Page iii 



Page iv 



Report Summary 



Background 



Federal Requirements 



Calculations of Graduate 
Data 



The Legislative Audit Committee prioritized a performance audit of 
Montana high school graduate and dropout rates to examine the 
requirements associated with the calculations and their use. The 
overall intent was to determine the effect and cause of any deviations 
from established requirements. The scope of this audit focused on 
three main activities: the current process of compiling data on 
graduate and dropout students, how the implementation of a new 
student information management system affects the process, and how 
graduate and dropout data is used. 

Although supervision and control of schools is dispersed among 
different entities, the Superintendent of Public Instruction 
administers the majority of services to students and teachers in over 
400 school districts, as well as compiles data from schools. Based 
on both state and federal requirements, the Office of Public 
Instruction (OPI)'s Division of Measurement and Accountability 
obtains data on student graduate and dropout numbers in the annual 
data collection, which schools/districts submit each fall. 

The federal No Child Left Behind Act (NCLB) (PL 107-1 10), 
enacted in 2002, holds states and the schools under their jurisdictions 
accountable for student performance. While student achievement 
must be the principal indicator of performance under NCLB, 
statewide accountability systems are also required to incorporate one 
additional academic outcome. At the secondary level, this "other 
academic indicator" is required by NCLB to be the high school 
graduate rate. States are required to set goals for themselves, and 
then each school and district is to meet these goals through various 
performance indicators. 

Although NCLB requires states to submit graduate data, it does not 

mandate how it is collected or calculated. The National Center for 

Education Statistics (NCES) suggests states use a Leaver-Based 

Graduation Rate. This definition follows groups of students 

throughout their four years of high school to calculate a graduation 

rate. Montana, along with 31 other states, currently uses this 
definition. 
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Calculations of Dropout 
Data 



School Visits 



Examples of Inconsistency 



NCLB does not require states to directly report dropout rates. 
However, the annual number of dropouts is needed to calculate the 
graduation rate submitted annually to the federal government. 
Montana collects dropout data from schools for local and state 
analysis as well. An annual dropout rate is used in Montana. This 
rate is established by dividing the annual enrollment by the number 
of dropouts in a given year. The NCES recommends this method for 
calculating dropout rates. It produces the lowest rate of any method 
since it is only a snapshot of one year. 

To assure accuracy and consistency between schools, OPI has 
established the Montana High School Completer and Dropout Data 
Collection Handbook. This handbook instructs schools on how to 
collect/input data to OPI for graduates and dropouts, as well as 
provides information on calculating dropout and graduation rates. 
To determine if these guidelines are followed we visited 14 schools 
around the state to observe and interview school personnel on how 
they use the OPI handbook and apply student status coding 
procedures. We compared OPI data to documentation maintained by 
the schools on the number of reported student graduates, dropouts 
and completers. In all 14 schools visited, 100 percent of the data 
reviewed at the schools was consistent to that reported by OPI. 

We found inconsistencies in the manner in which status codes were 
applied to students. During our visits we inquired how a student 
would be coded in different scenarios; what constitutes a graduate, 
completer, and dropout. The responses we received indicated 
schools have different interpretations on how and when to apply OPI 
student status codes as defined in guidelines. In 9 out of 14 schools 
visited, we found inconsistencies in the application of student status 
codes. For example, some schools categorize a student passing a 
General Education Development (GED) test as a completer, 
regardless of the administering entity. OPI, in accordance with 
NCLB, requires students to pass a GED test through a program 
administered by the school district in order to be coded as a 
completer. In most Montana districts, this is usually done through an 
alternative school within the district. A student who passes a GED 
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Quality Assurance System 
Needed 



Documentation Needed 
for District Approved 
GEDs and Home School 
Students 



OPI's Plan For Improving 
Data Collection and 
Accuracy 



test through another entity other than the school district should be 
coded a dropout, according to OPI guidelines. 

The National Center for Educational Accountability suggests, in 
addition to establishing a statewide coding system for student status, 
it is important to provide training on how those codes should be 
applied. OPI does not provide on-site technical assistance for 
Montana schools. There are no on-site visits to assist applicable 
school personnel in interpreting coding requirements or 
communicating guideline changes/updates. 

There are some limitations to OPI guidelines. OPI guidelines do not 
require verification of home school students and GED test recipients 
from the district. Although OPI's handbook states a home-schooled 
student or a GED test recipient within the district is not a dropout, 
audit work showed variation on the level of confirmation schools 
require for a student in these categories. In order for data to be 
accurate, OPI must be able to distinguish between students who drop 
out, get a GED, or who transfer to another school or home school 
situation. Montana school districts already require documentation of 
transcripts for students transferring among other public schools; the 
same standard could apply to the documentation of home schooling 
or the entity that issued a GED test to a student. 

OPI began exploring the creation of a comprehensive educational 
reporting system in 2003. The goal was to develop an infrastructure 
for the educational community to gather school data via the Internet, 
manage the secure storage of the data, and make the data accessible 
to decision makers throughout the educational community. In 2004, 
OPI contracted with an independent consultant to survey schools on 
the data currently collected and the systems they use to manage data, 
as well as write a concept and implementation plan for a student 
record system. One of the most important findings of the 
consultant's survey was that the majority of respondent districts were 
more than ready to see OPI move forward and initiate data 
improvements. 
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Resources Appropriated 



The New System 



In 2005, OPI requested funds to implement the student information 
management system. The legislature appropriated $2.8 million for 
the purpose of a new statewide K-12 education data system and 
approved four full time staff. Most recently, OPI assembled an 
evaluation team. They met on regular occasions, discussing and 
conducting research on similar systems and, eventually, discussing 
the request for proposal (RFP) for the creation and implementation 
of the new system. 

According to planning documents, the RFP, and interviews with OPI 
personnel, the new system will have three main components: 



Human Element Still 
Exists 



Data is Integral to 
Decision-Making 



► Student information, including student identifier and data 
elements, 

► Special education data, 

► An electronic data warehouse. 

OPI will begin initial implementation in the fall of 2006 by assigning 
a student identifier to each student enrolled in public school, K-12. 
OPI will still collect data on the number of graduates and dropouts, 
but with the student identification component it will be possible to 
roll up the data and look at overall numbers or drill down to see the 
status of one particular student. 

The implementation of the new student information system will 
improve OPI's current data collection and compilation system in 
various ways, but individuals will still decide on which status code to 
enter for a student. If individuals are not using student status codes 
consistently among schools, the new student information system will 
not provide comparable data. 

Educational stakeholders in Montana agree graduate and dropout 
data is an integral piece of information, specifically at the state level. 
They consider it to be an important tool in efficient decision-making 
and public reporting. Educational stakeholders believe the public 
has a right to know the number of students graduating and have 
access to that data. During audit work, one stakeholder discussed the 
importance of looking at the bigger picture and the economic 
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Data 



impacts of education. The importance of paying attention to students 
who are at risk of dropping out was highlighted. The number of 
individuals in prison and on social welfare without a degree is high. 
Stakeholders indicated the expense on the back end of "at-risk" kids 
is much more than if efforts would focus on the problem while they 
are in school. 



Accessibility and Use of The OPI Division of Measurement and Accountability, which 



collects, compiles, and analyzes graduate and dropout data, shares 
data with various programs within OPI and presents annually to the 
Board of Public Education. Some annual reports are posted to OPFs 
website as well. Audit work found no formal plan for educational 
stakeholders and the public to access and analyze data stored in the 
new system. The RFP does request the system have an interactive 
querying tool to enable data driven decision-making by state-level 
education staff and policy makers. Section 10.55.603, ARM, states 
the assessment of educational programs and their effectiveness 
should be examined through assessment results as well as graduate 
and dropout numbers. In order to examine graduate and dropout 
numbers for program effectiveness, statewide data needs to be 
accessible. A more comprehensive system that provides various 
means to access data would be more effective. 
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Introduction 



The Legislative Audit Committee prioritized a performance audit of 
Montana high school graduate and dropout rates to examine the 
requirements associated with the calculations and their use. The 
overall intent was to determine the effect and cause of any deviations 
from established requirements. For the 2004-2005 school year, the 
Office of Public Instruction (OPI) reported an 84.8 percent 
graduation rate and a 3.4 percent dropout rate. Audit work shows 
inconsistent procedures are used to categorize students at schools. 



Audit Scope and Objectives 



The scope of this audit focused on three main activities: the current 
process of compiling data on graduate and dropout students, how the 
implementation of a new student information management system 
affects the process, and how graduate and dropout data is used. 
Audit scope concentrated on graduate and dropout numbers in 
Montana high schools (grades 9-12) for the school year 2004-2005. 
Based on this scope, we established four main objectives: 



1 . Determine if guidelines and definitions provided by OPI are 
designed to create consistent, accurate data on high school 
graduates and dropouts. 

2. Determine if schools follow guidelines and definitions for 
reporting graduates and dropouts in a consistent manner. 

3. Determine if the new student information management system 
will provide comparable and uniform information on graduates 
and dropouts among Montana public schools. 

4. Determine if data on high school graduates and dropouts is used 
by educational stakeholders for the purpose of making informed 
decisions. 

These objectives were addressed through the following 
methodologies: 



Review of national educational policy centers' literature on 
graduate and dropout data. 

Review of state and federal laws applicable to the collection of 
graduate and dropout numbers. 

Review of legislative/agency documents and work completed to 
date on the new student information system. 
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► Visits to 14 high schools to interview school personnel, observe 
data collection, and verify aggregate data from OPI. 

► Review of other states' student information systems. 

► Interviews with educational stakeholders (education associations, 
schools, and Board of Public Education) on the use and 
importance of graduate and dropout data. 



Background 



State law separates responsibilities of the public school system 
among three entities. The Montana Constitution provides the Board 
of Public Education (BPE) with responsibility for long range 
planning, and coordination and evaluation of policies and programs 
of the public school system. Article X, Section 8, of the Montana 
Constitution vests the supervision and control of the schools in each 
district to an elected local board of trustees. Section 20-3-106, 
MCA, provides the Superintendent of Public Instruction with general 
supervision of public schools and districts of the state. Although 
supervision and control of schools is dispersed among different 
entities, the Superintendent of Public Instruction, administers the 
majority of services to students and teachers in over 400 school 
districts, as well as compiles data from schools. Based on both state 
and federal requirements, OPI's Division of Measurement and 
Accountability obtains data on student dropout and graduate 
numbers in the annual data collection (ADC), which schools/districts 
submit each fall. 



State Requirements 



Federal Requirements 



Section 10-55-603, ARM, established by the BPE, requires schools 
to "examine program effectiveness through graduation and dropout 
rates..." As a means to implement this rule, schools construct a five- 
year Comprehensive Education Plan which outlines goals for 
continuous educational improvements. Each district reports on its 
progress toward these goals annually in the ADC. 

The federal No Child Left Behind Act (NCLB) (PL 107-1 10), 
enacted in 2002, holds states and the schools under their jurisdictions 
accountable for student performance. While student achievement 
must be the principal indicator of performance under NCLB, 
statewide accountability systems are also required to incorporate one 
additional academic outcome. At the secondary level, this "other 
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academic indicator" is required by NCLB to be the high school 
graduation rate. States are required to set goals for themselves, and 
then each school and district is to meet these goals through various 
performance indicators. If a school does not meet the set graduation 
rate requirement, the district can incur fiscal sanctions under NCLB. 
Title I of NCLB provides funds to help improve instruction in high 
poverty schools and ensure poor and minority students have the same 
opportunity as other students to meet academic standards. Schools 
that consistently fail to meet these NCLB accountability 
requirements may be required to use Title I monies to fund 
supplemental education services, pay transportation costs for a 
student to attend another school within their district, and/or develop a 
restructuring plan. 



Calculations of Graduate Although NCLB requires states to submit graduate data, it does not 

^ mandate how it is collected or calculated. The National Center for 

Education Statistics (NCES) suggests states use a Leaver-Based 
Graduation Rate. This definition follows groups of students 
throughout their four years of high school to calculate a graduation 
rate. Montana, along with 31 other states, currently uses this 
definition. Other common graduation rates used nationally are 
detailed in Table 1 . 
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Table 1 
National Methods for Calculating Graduation Rates 



Rate 


Methodology 


Considerations 


Leaver-Based 
Graduation Rate 


Calculated by dividing number of high 
school diploma recipients by number 
of students known to leave school by 
dropping out, graduating or otherwise 
completing high school. 


■ Requires dropout data 
from four consecutive 
academic years, as well 
as number of graduates 
for the latest year 

■ Relies on accuracy of 
dropout data 

■ Does not use enrollment 


Enrollment-Based 
Cumulative Promotion 
Index 


This method assumes graduation is a 
process of three grade-to-grade 
transitions (9 to 10, 10 to 11, and 
11 to 12). Each transition is calculated 
as a probability by dividing the 
enrollment of the later year by the 
enrollment of the previous year. These 
separate probabilities are then 
multiplied to produce the probability 
that a student will graduate. 


■ Requires enrollment data 
from two consecutive 
academic years, as well 
as number of graduates 
for the latest year 

■ Requires calculation of 
district-level rates to 
determine the state rate 

■ Does not use dropout 
data 


Enrollment-Based 
Cohort Graduation Rate 


Calculated by dividing the number of 
students who receive a regular high 
school diploma by the number of 
students enrolled in ninth grade four 
years earlier. 


■ Requires enrollment data 
from five consecutive 
academic years, as well 
as number of graduates 
for the latest year 

■ Requires census 
population data 

■ Does not use dropout 
data 



Source: Data compiled by Standard and Poor's School Evaluation Services. 
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Calculations of Dropout NCLB does not require states to directly report dropout rates. 

Data 

However, the annual number of dropouts is needed to calculate the 

graduation rate submitted annually to the federal government. 
Montana collects dropout data from schools for local and state 
analysis as well. An annual dropout rate is used in Montana. This 
rate is established by dividing the number of dropouts by the number 
enrolled in a given year. The NCES recommends this method for 
calculating dropout rates. It produces the lowest rate of any method 
since it is only a snapshot of one year. 

There are three main methods used by states to calculate dropout 
rates. 



Page 5 



Chapter I - Introduction and Background 



Table 2 
Methods for Calculating Dropout Rates 





Annual Dropout Rate 


Longitudinal Dropout Rate 


Attrition Rate 


Description 


Measures the percentage of 
students who drop out of 
school during one year. 


Measures the percentage of 
students from a class of 9* 
graders who drop out before 
completing high school. 


Estimates the percentage of 
students from a class of 9" 
graders not enrolled in 
grade 12 four years later 
for whatever reason. 


Calculation 


Divide the number of 
students who drop out 
during a school year by the 
total number enrolled that 
year. 


Divide the number of students 
who drop out by the end of 
grade 12, by the total number of 
students in the original 9"grade 
class. Students who transfer in 
or out over the years are added 
or subtracted. 


Subtract grade 12 
enrollment from grade 9 
enrollment four years prior 
then divide by the grade 9 
enrollment. 


Advantages 


• Measures annual 
performance 

• Requires only one 
year of data 

• Can be calculated for 
any school or district 
with students in any 
of the grades covered 

• Can be disaggregated 


• More consistent with the 
publics' understanding of 
a dropout rate 

• Districts have more time 
to encourage dropouts to 
return to school before 
being held accountable 


• Provides a simple 
measure of school 
leavers when 
aggregate enrollment 
numbers are all that 
is available 


Disadvantages 


• Produces the lowest 
rate of any method 

• May not correspond 
to the public's 
understanding of a 
dropout rate 


• Requires multiple years 
of data 

• Does not produce a 
dropout rate by grade 

• Program improvements 
may not be reflected for 
several years 


• Produces the highest 
rate of any method 

• Does not distinguish 
attrition resulting 
from dropping out 
from that resulting 
from transfers, 
retentions, death, etc. 


School example 
(Grades 9-12) 


2.2% 


8.5% 


36.6% 



Source: Compiled by the Legislative Audit Division. 



The table defines and discusses the advantages and disadvantages of 
each rate. The bottom row of the table depicts a dropout rate for the 
same school, same year, for each calculation. It shows the 
differences produced among the methods. 
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Recent national studies comparing state graduation rates found 
different methods produce different results and in some cases the 
disparities can be quite large. More specifically, national research 
suggests that approaches for calculating graduation rates based on 
dropout data produce inflated graduation estimates compared with 
other methods. This is due to the reliance on the accuracy of dropout 
data, and not using enrollment data as a basis but rather diploma 
recipients. Currently, state accountability plans submitted for 
NCLB, use graduation rates that incorporate dropout data in more 
than half of all states (including Montana). Changes and 
improvements in Montana's methods are discussed in Chapters III 
and IV. 



National Attention The calculation of graduation and dropout rates has received national 

attention due to the high standards set by NCLB's accountability 
requirements. Through audit work, various states have discovered 
inaccuracies in their calculation of graduates and dropouts. The 
following illustrates some examples: 

► A Texas audit of 1 6 Houston schools found that more than half 
of the 5,500 students who left in the 2000-2001 school year 
should have been declared dropouts, but were not. That year, 
Houston schools reported that only 1.5 percent of its students 
dropped out. 

► A 2001 Michigan audit of the educational accountability 
program found the education department did not have an 
effective process for gathering data, and that calculations used 
for graduate and dropout rates were not in accordance with state 
statutes. 

The national attention on misreporting of graduate and dropout 
numbers spurred educational policy groups, including a National 
Governors Association Task Force, to recommend the creation and 
establishment of better systems and methods of collecting, analyzing, 
and reporting graduate and dropout data, which includes looking at 
longitudinal data for both groups. The difficult part of this task is 
accurately accounting for students who do not finish, and 
distinguishing between those who should be counted as dropouts and 
those who should not. In order to calculate precise graduation rates. 
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states need integrated information systems that can tell if a student 
who stopped attending one high school shows up in another school. 



Report Organization The next chapter will outline how OPI currently collects and guides 

schools on graduate and dropout data. Chapter III details OPFs 
current plans for improving data collection, and the last chapter 
discusses the importance of using dropout and graduate data for 
decision-making . 



Pages 



Chapter II - OPI Guidelines 



OPI Guidelines for 
Reporting 



The Office of Public Instruction (OPI) is required to submit Montana 
graduation rates annually to the federal government for No Child 
Left Behind (NCLB) requirements. To assure accuracy and 
consistency between schools, OPI has established the Montana High 
School Completer and Dropout Data Collection Handbook. These 
guidelines were developed in the early 1990's, and are continually 
updated, taking language directly from the National Center on 
Education Statistics. These guidelines outline definitions for 
dropouts, graduates, completers, and no shows. This handbook 
instructs schools how to collect/input data to OPI for graduates and 
dropouts, as well as provides information on calculating dropout and 
graduation rates. OPI personnel calculate these percentages. Listed 
in the following table are brief definitions for the six OPI-defined 
categories of Montana student status. 
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Table 3 
Student Status Definitions 



Student Status 


Definition 


Graduate 


A student who completed the high school graduation 
requirements of a school district in four years or less, 
or had a special education plan allowing more than four 
years to graduate. 


Completer 


A student who passes a GED test through a program 
administered by the school district and/or a student 
who takes more than the standard number of years (4) 
to receive a high school diploma. 


No Show 


A student who was enrolled in school the previous 
year, but as of the fall collection date, the student has 
not shown up to school. The student should be counted 
as a dropout the following year if a transfer status is not 
confirmed. 


Dropout 


A student who was enrolled the previous year but quit 
school at some point and is not currently enrolled. 


Transfer 


A student who exits from one school to enroll in 
another. 


Home School 


A student who has left the public school system to be 
taught at home and is registered with the county 
superintendent as a home schooler. 



Source: Compiled by the Legislative Audit Division from agency records. 



In defining student status, the handbook provides a different example 
and perspective to illustrate each definition. According to the 
National Center for Educational Accountability (NCEA), one of ten 
essential elements in collecting quality data is having accurate 
student level dropout and graduate data. In an effort to do this, 
NCEA suggests establishing a statewide coding system for the 
various reasons students leave a school and clear guidelines on how 
those codes should be applied. OPI has established such a system for 
Montana. 



School Visits 



To determine if these guidelines are followed, we visited 14 schools 
around the state to observe and interview school personnel on how 
they use the OPI handbook and apply status coding procedures. The 
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Data Verification 



Examples of Inconsistency 



schools were chosen based on their average enrollment and dropout 
rate. Averages for both enrollment and dropout rates were calculated 
using three years of data (2001-2004). We divided Montana high 
schools into four groups - high, medium, low and zero percent 
dropout rates. Within the four groups, we chose schools based on 
enrollment size and dropout rate. A small, medium and large 
enrollment school was chosen within each of the four dropout 
groups. 

We compared OPI data to documentation maintained by the schools 
on the number of reported student graduates, dropouts and 
completers. In all 14 schools visited, 100 percent of the data 
reviewed at the schools was consistent to that reported by OPI. Our 
review confirmed that the data in school information systems 
matched data input into OPI's system. 

We found inconsistencies in the manner in which status codes were 
applied to students. During our visits we inquired how a student 
would be coded in different scenarios; what constitutes a graduate, 
completer, and dropout. The responses we received indicated 
schools have different interpretations on how and when to apply OPI 
defined status codes. In 9 out of 14 schools visited, we found 
inconsistencies in the application of student status codes. For 
example, some schools categorize a student passing a General 
Education Development (GED) test as a completer, regardless of the 
administering entity. OPI guidelines, in accordance with NCLB, 
require students to pass a GED test through a program administered 
by the school district in order to be coded as a completer. In most 
Montana districts, this is usually done through an alternative school 
within the district. A student who passes a GED test through another 
entity other than the school district should be coded a dropout, 
according to OPI guidelines. 



Another example is a student that attends Job Corps. Depending on 
the school, this type of student is coded in various ways; dropout, 
graduate, and completer. Job Corps is an education and vocational 
training program administered by the U.S. Department of Labor that 
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serves young people ages 16 through 24. At Job Corps, students 
enroll to learn a trade, earn a high school diploma/GED, and get help 
finding a job. OPI's guidelines are unclear in defining a status code 
for Job Corps students or when those students should be coded as a 
dropout versus a completer. 

Another variation found in schools was how a "no show" student is 
coded. A "no show" student is one who was enrolled either from 
eighth grade or in the previous year; however, when school begins in 
the fall, this student does not show up. Some schools are reporting 
this student as a dropout in the Annual Data Collection (ADC) the 
same year, while others are waiting until the following school year to 
report, in hopes of receiving more information on whether the 
student is a dropout or transfer. OPI guidelines allow a school to 
wait until the following year to report a "no show." Our visits also 
found a school that only reports dropouts as those students who drop 
out of the twelfth grade, rather than grades 9-12, as required. 
Although ethnicity has no effect on dropout or graduation numbers, 
we found a school that coded students as White, non-Hispanic, even 
when that was not the student's ethnicity. 



Guidelines Not Fully Utilized 



Although OPI's dropout and graduate handbook provides detailed 
guidelines for student status codes, schools are using the guidelines 
on a limited basis. As part of our audit work, we interviewed 
officials responsible for tracking student status and reporting it to 
OPI. The staff person responsible for these duties varied by school 
and included either the secretary, principal, or data specialist. 
During interviews we asked if these officials found the handbook 
useful. Three of the 14 schools we visited found OPI guidelines for 
graduates and dropouts to be very useful. Eight schools use the 
guidelines to some degree and the other three do not use the 
guidelines at all. Therefore, a majority (1 1 of 14) of the schools are 
using the guidelines on a limited basis or not at all. 



A Quality Assurance System 
Needed 
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NCEA suggests, in addition to establishing a statewide coding 
system for student status, it is important to provide training on how 
those codes should be applied. They also recommend each state 
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have a system for occasionally spot-checking the accuracy of data in 
cases that are not flagged by statistical checks. Currently, OPI has 
not developed this level of quality assurance. For example, OPI does 
not provide on-site technical assistance for Montana schools to assist 
applicable school personnel in interpreting coding requirements or 
communicating guideline changes/updates. 



Conclusion 



OPI's Montana High School Completer and Dropout Data 
Collection Handbook is an effort to create consistent and accurate 
data among schools. The handbook provides clear, detailed 
guidelines. However, when establishing policy (handbook) the 
issuing entity has a responsibility to ensure it is communicated and 
followed by the intended recipients (schools). Additional assurance 
could be provided through OPI staff conducting on-site audits of 
status code application, as well as providing technical assistance. 
Off-site assurances could include building on existing statistical 
checks in the database, as well as randomly calling to verify a 
student who has recently changed status. 



Recommendation #1 



We recommend the Office of Public Instruction establish a 
quality assurance system that includes: 

A. On-site technical assistance training on status code 
procedures and requirements. 

B. Random verification of student status coding and data 
submitted. 



Documentation Needed 
for District Approved 
GEDs and Home School 
Students 



There are some limitations to OPI guidelines. OPI guidelines do not 
require verification of home school students and GED test recipients 
from the district. Although OPI's handbook states a home-schooled 
student or a GED test recipient within the district is not a dropout, 
audit work showed variation on the level of confirmation schools 
require for a student in these categories. 



Home School Students 



Home school students in Montana are required to register with the 
County Superintendent of Schools. Some of our sample schools 
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verify the student's registration with the County Superintendent 
while other schools do not. OPI guidelines do not require schools to 
verify home school registration. During interviews with school 
personnel, concerns of schools misclassifying a student as a 
homeschooler arose. School personnel stated some other schools 
may be coding a dropout as a home school student in order to not 
negatively affect their school's graduation and dropout rates. By not 
accurately coding a dropout, schools inflate their graduation rate and 
decrease their dropout rate. 



District Approved GED 
Recipients 



The level of confirmation varies on students who pass the GED test 
as well. As mentioned earlier, we found schools did not confirm 
whether a GED test was taken through a district-administered 
program. Audit work found some schools categorize a student 
completing a GED, even if outside the district (not district 
administered), as a completer. In one example a student passed a 
GED test through a private program after dropping out of school. 
School personnel became aware of the student's GED achievement, 
and therefore coded the student' s status as a completer instead of a 
dropout. OPI requires students to receive a GED from a 
district-administered program, which is usually an alternative school 
within the district, in order to be counted as a completer. This 
requirement is consistent with NCLB law. 



In order for data to be accurate, OPI must be able to distinguish 
between students who drop out, get a GED, or who transfer to 
another school or home school situation. Montana school districts 
already require documentation of transcripts for students transferring 
among public schools; the same standard could apply to 
documentation of home schooling or the entity issuing a GED test to 
a student. 



Recommendation #2 



We recommend OPI incorporate into its guidelines that schools 
verify and document home school registration and GED test 
information. 
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Collection and Accuracy 



Initial Planning 



The Office of Public Information began exploring the creation of a 
comprehensive educational reporting system in 2003. The goal was 
to develop an infrastructure for the educational community to gather 
school data via the Internet, manage the secure storage of the data, 
and make the data accessible to decision makers throughout the 
educational community. In 2004, OPI contracted with an 
independent consultant to survey schools on the data currently 
collected and the systems they use to manage the data, as well as 
write a concept and implementation plan for a student information 
system. 



Resources Appropriated 



Current Stage of 
Implementation 



One important finding of the consultant's survey was the majority of 
respondent districts were more than ready to see OPI move forward 
and initiate data improvements. Many districts indicated they were 
already collecting student level data and could provide such data to 
OPI in many of the areas surveyed. In the concept and 
implementation plan, the consultant emphasized the need for OPI to 
provide leadership, coordination and support for a statewide student 
information system. The plan states a comprehensive reporting 
system should be the primary source of educational information. 
The consultant's work provided OPI with the feedback necessary to 
move forward on implementing the student information system. 

In 2005, OPI requested for funds to implement a student information 
system. The legislature appropriated $2.8 million for the purpose of 
a new statewide K-12 system and approved four full-time staff. The 
request for proposal (RFP) for the system went out with a cost 
estimate of $2 million, which leaves $800,000 for the new positions 
over the 2007 biennium. Positions being hired for this project are: 
database administrator, project manager, student records manager, 
and data resource administrator. 

Most recently, OPI assembled an evaluation team which was 
comprised of eight OPI staff members, as well as two school staff 
members with knowledge about information technology and special 

Page 15 



Chapter III - OPI's Plan For Improving Data Collection and Accuracy 

education. They met on regular occasions, discussing and 
conducting research on similar systems for the creation and 
implementation of the new system. Research consisted of important 
components to consider such as: 

► An audit system to assess data quality, validity, and 
reliability. 

► A unique identifier for each student in the system. 

► Interoperability. 

► Privacy protection. 



The New System OPI is currently in the process of contracting with a vendor to 

develop the new system. According to planning documents, the 
RFP, and interviews with OPI personnel, the new system will have 
three main components: 

► Student information; including student identifier and data 
elements, 

► Special education data, 

► An electronic data warehouse. 

The student identification component will provide a unique identifier 
for each student entering the Montana public education system. This 
number will remain the student's identifier until he/she exits the 
system. As well, it will provide information on each student such as 
gender, grade, etc. Currently, there is no way to track students 
throughout the public education system because OPI collects 
aggregate numbers, not information to identify or track an individual 
student through the education system. 

The second main component will provide information on the 
statewide system for special education students. All documents 
pertinent to the student will be accessed electronically through the 
student's identification number. 

The third component of the new system will be an electronic data 
warehouse where all student/school level data will be stored 
annually. This data will be updated periodically as student 
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information changes. Information can be accessed to conduct trend 
analysis by district or state personnel. 



Majority of States Have a 
Statewide Information 
Management System 



OPI will begin initial implementation in the fall of 2006 by assigning 
a student identifier to each student enrolled in public schools, K-12. 
OPI will still collect data on the number of graduates and dropouts, 
but with the student identification component it will be possible to 
roll up the data and look at overall numbers or drill down to see the 
status of one particular student. 

A student identifier and data warehouse system similar to what 
Montana is implementing exists in over half of the states. Thirty-two 
states and the District of Columbia already have such a system in 
place. Sixteen states, including Montana are currently developing a 
system. Only two states have not yet looked at developing such a 
system. 



The National Center for Educational Accountability (NCEA) 
recommends states institute a longitudinal data system which 
matches individual student records over time, from pre-kindergarten 
through twelfth grade and into post-secondary education. NCEA 
states, 

"That when states collect the most relevant data and are able 
to examine longitudinally, they can answer the questions 
that are at the core of educational effectiveness. States are 
spending hundreds of millions of dollars to improve student 
achievement. But without quality data, they are essentially 
flying blind." 

Other states have various components to their statewide student 
information management systems; for example, teacher information, 
fiscal or budget information, and post-secondary information. 
Although OPI documents and interviews suggest a desire for these 
additional components, at this point, there is only funding allocated 
to implement the three components previously mentioned. 
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Impacts of the New 
System 



The most obvious benefit fi^om the graduate and dropout perspective 
is the ability to track or follow students within Montana public 
schools. Currently, one student could be coded as a dropout fi^om 
two schools in the same year. If a student attends two schools and 
drops from both within one year, duplications in student coding 
occur. If a student transfers to another school, but the receiving 
school never requests transcripts, the student may be coded as a 
dropout rather than a transfer. The new system will help address 
these problems. No longer will OPI's data be just disaggregated 
numbers; with the new system there will be a higher level of detail 
available. For example, stakeholders will be able to correlate certain 
factors such as the number of dropouts to students in advanced 
placement or special education classes statewide. Because there will 
be a data warehouse component to the system, longitudinal data or 
trend analysis will be possible. In addition, stakeholders believe a 
longitudinal rate for dropouts would be a better representation of the 
dropout rate for Montana. 



Human Element Still Exists 



The implementation of the new student information system will 
improve OPI's current data collection and compilation system in 
various ways, but individuals will still decide on which status code to 
enter for a student. If individuals are not using student status codes 
consistently among schools, the new student information system will 
not provide comparable data. The importance of this was stressed in 
Chapter II and addressed in Recommendation 1. 



The issue was also raised in various interviews with stakeholders. 
The subject of consistent, accurate data came up repeatedly. 
Stakeholders do not believe all schools are defining a dropout in the 
same manner. Our school visits reaffirm this statement. Consistency 
in the application of student status codes is essential to assure the 
validity and accuracy of the data. 
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Conclusion: The statewide student information system will 
assist in providing quality, uniform data on graduates and 
dropouts. However, OPI's data collection system will still 
largely depend on the consistent application of student status 
codes by school personnel. 



Chapter IV - Use of Data 



Data is Integral to 
Decision-Making 



Educational stakeholders in Montana agree graduate and dropout 
data is an integral piece of information, specifically at the state level. 
They consider it to be an important tool in efficient decision-making 
and public reporting. Educational stakeholders believe the public 
has a right to know the number of students graduating and have 
access to that data. During audit work, one stakeholder discussed the 
importance of looking at the bigger picture and the economic 
impacts of education. The importance of paying attention to students 
who are at risk of dropping out was highlighted. The number of 
individuals in prison and on social welfare without a degree is high. 
The expense on the back end of "at-risk" kids is much more than if 
efforts would focus on the problem while they are in school. 



National Use of Data for 
Economic Analysis 



The National Center for Education Statistics, in conjunction with the 
U.S. Department of Commerce, confirms the importance of tracking 
these areas, by stating, "Dropouts from high school are more likely 
to be unemployed and earn less when they are employed than those 
who complete high school. " Research presented to the New Mexico 
Business Roundtable for Educational Excellence by the Office of 
Education Accountability estimates a lifetime loss of tax revenues, 
from males ages 25-34 years who did not complete high school, to be 
approximately $944 billion, with cost increases to public welfare and 
crime at $24 billion. 



Montana decision-makers could use similar approaches to look at the 
effects of dropouts on Montana's economy. The following chart 
depicts one method of calculating the economic impacts of a high 
school diploma in Montana. 
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Figure 1 

Economic Benefits of Education in Montana 
Annual Increase in Earnings by Educational Level 
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Source: Data compiled by Alliance for Excellent Education, 2005. 



Stakeholders' Use of Data 



The increased earnings in the above chart were calculated using 
graduate rate data from the National Center for Education Statistics 
and the Manhattan Institute for Policy Research and a formula from 
the U.S. Department of Commerce. The availability of better 
graduate and dropout data provides a better means to analyze 
economic costs and benefits of education. 

At the local level, schools use graduate and dropout data to assess the 
performance of their individual school and district. Administrators 
review the data collected on the number of graduates and dropouts in 
order to discern the bigger picture of the school's performance. One 
educational stakeholder stated, "dropout and graduate data is just one 
indicator of school performance, but a foundational one so the 
district and state know where to put their resources." A 
superintendent of a large district in the state developed dropout 
prevention programs and started an alternative high school because 
the data for his district showed high numbers of dropouts. Because 
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graduation and dropout rates are performance indicators for schools, 
stakeholders indicated they would like to see a more consistent 
depiction of the rates. 



Accessibility and Use of The OPI Division of Measurement and Accountability at the Office 

Data of Public Instruction collects, compiles, and analyzes graduate and 

dropout data, shares data with various programs within OPI and 
presents annually to the Board of Public Education. Upon request, 
the division also provides data to educational stakeholders in the 
state and researchers from around the nation. Some annual reports 
are posted to OPI's website, as well. Audit work found no formal 
plan for educational stakeholders and the public to access and 
analyze data stored in the new system. The RFP does request the 
system have an interactive querying tool to enable data driven 
decision-making by state-level education staff and policy makers. 

In the OPI concept and implementation plan for the new statewide 
student information system the term "information democracy" is 
used. Information democracy means everyone (with appropriate 
security) can access data in the data warehouse component of the 
system (i.e., it is an open system). Different stakeholders in the 
educational community need different types of access to data. For 
example, members of the general public often need summary or 
"canned reports" from the data warehouse, whereas some researchers 
or legislators may need to conduct a series of 'ad hoc' queries. In 
such a system, less experienced data users must have reports 
regularly pushed to them via the Internet and those groups that have 
the capability for greater analysis may have permission to pull more 
selective data at any time from the data warehouse. 

Section 10.55.603, ARM, states the assessment of educational 
programs and their effectiveness should be examined through 
assessment results as well as graduate and dropout numbers. In 
order to examine graduate and dropout numbers for program 
effectiveness, statewide data needs to be accessible. A 
comprehensive system that provides various means to access data 
would be more effective. 
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Conclusion 



OPI is working to implement a $2 million statewide student 
information system. The implementation of a new information 
system would be an optimal time to establish a plan for data 
accessibility and dissemination. An accessibility and distribution 
plan would enable the state to effectively utilize the $2 million 
information system to its full potential. The implementation of the 
new system is the first step to providing quality data to educational 
stakeholders. However, this data should be used and analyzed. 
A plan for data distribution and accessibility is the next step in fully 
utilizing the student information management system. 



Recommendation #3 



We recommend OPI establish a plan for accessing and 
disseminating the data in the new student information system. 
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Appendix A - Average Dropout Rate 

By High School 



School 

Absarokee High School 
Alberton High School 
Anaconda High School 
Arlee High School 
Augusta High School 
Bainville High School 
Baker High School 
Beaverhead Co High School 
Belfry High School 
Belgrade High School 
Belt High School 
Big Sandy High School 
Big Sky High School 
Bigfork High School 
Bilhngs Sr High School 
Bilhngs West High School 
Bigs Central Catholic H S 
Blue Sky High School 
Box Elder High School 
Bozeman High School 
Brady High School 
Bridger High School 
Broadview High School 
Broadwater High School 
Brockton High School 
Browning High School 
Butte Central High School 
Butte High School 
C M Russell High School 
Capital High School 
Carter County High School 
Cascade High School 
Centerville High School 
Charlo High School 
Chester High School 
Chinook High School 
Choteau High School 
Circle High School 
Colstrip High School 
Columbia Falls High Schl 



3 Year Average 
Dropout Rate 

0.53% 
1.33% 
3.62% 
2.64% 
0.00% 
0.00% 
0.37% 
2.63% 
3.23% 
4.28% 
1.33% 
1.33% 
3.16% 
2.56% 
4.67% 
3.02% 
0.40% 
0.00% 
12.32% 
3.22% 
3.09% 
1.00% 
0.74% 
3.00% 
3.35% 
7.25% 
0.22% 
5.58% 
2.18% 
3.32% 
0.99% 
3.24% 
1.79% 
2.54% 
1.12% 
2.77% 
0.40% 
1.19% 
2.09% 
4.29% 
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School 



3 Year Average 
Dropout Rate 

Columbus High School 1 .70% 

Conrad High School 3.14% 

Corvallis High School 3.98% 

Culbertson High School 0.47% 

Custer Co District High 4.15% 

Custer High School 1.10% 

Cut Bank High School 4.01% 

Darby High School 5.50% 

Dawson High School 2.13% 

Denton High School 1 .24% 

Dodson High School 0.00% 

Drummond High School 0.00% 

Dutton High School 0.00% 

Eagle High School 0.00% 

Ennis High School 0.00% 

Fairfield High School 0.00% 

Fairview High School 3.45% 

Fergus High School 3.03% 

Flathead High School 7.47% 

Flaxville High School 0.00% 

Florence-Carlton HS 0.85% 

Forsyth High School 3.25% 

Fort Benton High School 1.16% 

Frazer HighSchool 7.52% 

Frenchtown High School 2.84% 

Froid High School 0.00% 

Fromberg High School 2.33% 

Gardiner High School 2.13% 

Garfield Co Dist HS 0.96% 

Geraldine High School 0.75% 

Geyser High School 0.00% 

Glasgow High School 2.58% 

Granite High School 4.33% 

Grass Range High School 0.64% 

Great Falls Cent Cath HS 0.66% 

Great Falls High School 3.10% 

Hamilton High School 3.42% 

Hardin High School 5.74% 

Harlem High School 6.10% 

Harlowton High School 1 .25% 

Harrison High School 0.00% 

Havre High School 3.95% 
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School 



3 Year Average 
Dropout Rate 

Hays-Lodge Pole High Sch 1 1.56% 

Heart Butte High School 12.68% 

Helena High School 5.43% 

Hellgate High School 5.10% 

Highwood High School 0.82% 

Hinsdale High School 0.00% 

Hobson High School 0.00% 

Hot Springs High School 0.86% 

Huntley Project High Schl 0.88% 

Hysham High School 1 .27% 

Jefferson High School 2.44% 

Joliet High School 1.12% 

Joplin-Inverness HS 0.00% 

Judith Gap High School 0.00% 

Kalispell Jr High School 9th 1 .62% 

K-G High School 0.00% 

Lambert High School 0.00% 

Lame Deer High School 10.88% 

Laurel High School 3.57% 

Lavina High School 0.00% 

Libby High School 4.21 % 

Lima High School 0.00% 

Lincoln Co High School 2.58% 

Lincoln High S chool 3.33% 

Lodge Grass High School 18.71% 

Loyola-Sacred Heart H S 0.16% 

Lustre Christian H S 0.00% 

Malta High School 0.35% 

Manhattan Christian H S 0.00% 

Manhattan High S chool 0.96% 

Medicine Lake High School 0.00% 

Melstone High School 1.19% 

Moore High School 1 .05% 

Mount Ellis Academy High 0.44% 

MT Sch For Deaf & Bind HS 0.00% 

Nashua High School 0.00% 

Northern Cheyenne Trib HS 8.46% 

Noxon High School 1 .67% 

Opheim High School 0.00% 

Outlook High School 0.00% 

Park City High School 0.00% 

Park High School 2.54% 
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School 



3 Year Average 
Dropout Rate 

Peerless High School 0.00% 

Pine Hills School HS 0.00% 

Plains High School 2.56% 

Plenty Coups High School 1 1 .06% 

Plenty wood High School 2.12% 

Plevna High School 0.00% 

Poison High School 6.93% 

Poplar High School 10.17% 

Powder River Co Dist High 0.23% 

Powell County High School 6.86% 

Power High School 2.50% 

Rapelje High School 0.00% 

Red Lodge High School 1.53% 

Reedpoint High School 0.00% 

Richey High School 0.00% 

Riverside High School 0.00% 

Roberts High School 0.00% 

Rocky Boy High School 12.18% 

Ronan High School 8.38% 

Rosebud High School 1.15% 

Roundup High S chool 4.11% 

Roy High School 0.00% 

Ryegate High School 2.78% 

Saco High School 1.10% 

Savage High School 1.56% 

Scobey High School 0.30% 

Seeley-Swan High School 2.56% 

Sentinel High School 3.28% 

Shelby High School 7.55% 

Shepherd High School 3.19% 

Sheridan High School 1.10% 

Shields Valley High Schl 2.47% 

Sidney High School 2.57% 

Simms High School 3.53% 

Skyview High School 1.96% 

St Ignatius High School 4.94% 

St Labre High School 2.52% 

St Regis High School 1 .00% 

Stanford High School 1.19% 

Stevensville High School 3.93% 

Sunburst High School 1 .49% 

Superior High School 1 .46% 
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School 



3 Year Average 
Dropout Rate 

Sweet Grass Co High Schl 2.38% 

Terry High School 1.56% 

Thompson Falls High Schl 2. 14% 

Three Forks High S chool 1 .43 % 

Troy High School 1.27% 

Turner High School 1.10% 

Twin Bridges High School 0.46% 

Two Eagle River High Schl 8.26% 

Valier High School 0.44% 

Victor High School 3.24% 

West Yellowstone HS 6.82% 

Westby High School 0.00% 

White Sulphur Springs HS 3.80% 

Whitefish High School 3.05% 

Whitehall High School 3.39% 

Whitewater High School 0.00% 

Wibaux High School 0.99% 

Willow Creek High School 0.00% 

Winifred High School 0.72% 

Winnett High School 0.00% 

Wolf Point High School 5.68% 
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— Montana Office of Public Instruction 



Linda, McCulioch 

State Superintendent 



June 1,2006 RECEIVED 

JUN 5 2U06 

Scott A Seacat 

Legislative Auditor LEGiSLATiVE AUQH DIV. 

Legislative Audit Division 

Room 135, State Capitol Building 

P.O. Box 201705 

Helena, MT 59620-1705 

Dear Mr. Seacat: 

The purpose of this letter is to provide comments by the Office of Public Instruction (OPI) 
concerning the Legislative Audit Division's performance audit of the collection and use of 
dropout and graduate data. We thank Angle Grove and Bridget Butler for their hard work and 
professionalism during this audit. The following is our response to the recommendations in the 
performance audit. As you know, the Office of Public Instruction is currently implementing a 
statewide student information system with funding provided by the 2005 legislature. Many of 
our comments reflect the plans that OPI has made for working with a vendor to implement the 
student information system. 

Recommendation #1 

We recommend the Office of Public Instruction establish a quality assurance system that 

includes: 

A. On-site technical assistance on status code procedures and requirements. 

OPI Response: We concur. OPI understands the importance of providing high quality, on 
going training to school personnel. The new student information system includes a 
comprehensive training component on the use of the student information system, including the 
management of student enrollment status codes. OPI will use a standard set of codes and instruct 
school personnel on the appropriate use of these codes. OPI's contract with the software vendor 
calls for training of school personnel to be provided for at least two days in five regions in 
Montana. Given that school districts will be expected to maintain these codes on a regular basis 
(rather than for the purpose of an annual report), the more frequent usage should improve the 
quality of the data. 

Currently, OPI provides regional training each fall to school personnel on reporting non-fiscal 
data through the Annual Data Collection. At this training, OPI refers districts to the guidance 
provided in the OPI's Montana High School Completer and Dropout Data Collection Handbook. 
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Attendance at these training sessions is voluntary, and the amount of training time allocated to 
dropout and graduate reporting is approximately 10 minutes with additional time for follow-up 
questions. 

OPI does not have the financial resources to conduct individual school visits on a regular basis. 
Due to limited training budgets, travel budgets and staff resources, OPI must provide its 
technical assistance through regional trainings and electronic formats. In years past, OPI was 
able to conduct on-site accreditation reviews and provide assistance to districts for curriculum 
development. If additional funding were provided for on-site school visits, the priority for 
technical assistance would be in the area of academic program improvements and curriculum 
development. 

B. Random verification of student status coding and data submitted. 

OPI Response: We concur. Each June, OPI provides a set of instructions to school district 
auditors for items to review in the audit process. Currently, auditors are instructed to verify 
enrollment records. OPI could amend its instructions to auditors to include a review of the 
enrollment status of students. OPI could also generate a random list of school districts for 
auditors to conduct a more detailed check of the enrollment status of students. 

Under the new student information system, schools will be expected to maintain student 
enrollment status codes and will receive training on how to do so. OPI will have quality controls 
in place to ensure that the enrollment status of each student is valid. For example, a student 
could not be reported as enrolled in one district and as a dropout in another district. In addition, 
a district cannot report a student as having transferred to another public school district without 
that student showing up as enrolled in the public school system. 



Recommendation #2 

We recommend the Office of Public Instruction incorporate into its guidelines that schools 

verify and document home school registration and GED test information. 

OPI Response: We concur with the need to have systems in place to document and verify 
changes in a student's enrollment status. Beginning in the 2006-2007 school year, each student 
who is enrolled in a public school district will have an enrollment status. For example, a student 
could be coded as currently enrolled, transferred, dropped out, completed, not enrolled (but 
eligible to return), or deceased. As the student's enrollment status changes, the changes will be 
documented and saved to create a historical record for the student. 

When a student transfers from one public school district to another, the request for transcripts 
comes from the receiving district not from the sending district. In the case of a student 
transferring to a home school, the request for transcripts would be made by the parent. Under the 
new system, OPI could design a form that the sending school could ask the parent to sign when 
the student is withdrawn from the public school. However, there is nothing in state law that 
would require the parent to sign the form. County superintendents could also play a role in 
communicating to public school districts when a student has transferred to a home school. 
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Parents who home school their children are required to notify the county superintendent that the 
student is attending home school. 

GED students who are in a district-sponsored program are considered enrolled in school up until 
the time they take the GED test. In which case, there is no need for a transcript request because 
technically the student has not left the school or district. When an enrolled student completes 
their education by passing a GED test, the enrollment status code will reflect the student's 
completion status. If the student leaves school without passing the GED test, the student would 
be considered a dropout. 



Recommendation # 3 

We recommend that OPI establish a plan for accessing and disseminating the data in the 

new student information system. 

OPI Response: We concur. One of the critical success factors that OPI has identified for data 
warehouse and student information system is "providing state, school district, school, teacher, 
parent, and public access to appropriate education data, reports and decision support tools 
through a flexible, easy-to-use Web interface." The contract with the vendor includes both the 
creation of standard reports that can be generated on the Web in real time and the generation of 
ad-hoc reports with custom data elements. One of the significant benefits of the data warehouse 
is that the policy-makers and the public will have access to data once the data has been verified 
and that analysis tools will be provided for looking at longitudinal data and trends. 



Additional Information 

The table below provides additional information to accompany this audit report to show the 
numbers of students that were reported as standard graduates, non-standard graduates, other 
completers, and 1 2"^ grade dropouts for the 2004-2005 school year. 



Class of 2005 (Montana Public and State Funded Schools) 


Student Status 


Definition 


Student Counts 


Standard Graduate 


A student who completed the 
high school graduation 
requirements of a school 
district in four years or less, or 
had a special education plan 
allowing more than four years 
to graduate. 


10,272 


Non-standard Graduate 


A student who completed the 
high school graduation 
requirements of a school 
district in more than four years. 


63 


Other Completer 


A student who passes a GED 
test while enrolled in a program 
administered by a school 


82 
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district. 




Dropout (12"" grade) 


A student who was enrolled the 
previous year but quit school at 
some point and is not currently 
enrolled. This count includes 
students who are working on a 
GED in a program not 
administered by a school 
district. 


455 



Thank you for the opportunity to comment on the performance audit report. We look forward to 
the support from the Legislature to fully implement our student information system and the 
recommendations in this report. 




IcCulloc 

State Superintendent 




Page B-6 



